Generating a 3D shape template of a moving and deforming object from an RGB-D image sequence

Automatically reconstructing a 3D shape model of a nonrigid object using a sequence from a single commodity RGB-D sensor is a challenging problem. Some techniques use a 3D shape template of a target object; however, in order to generate the template automatically, the target object required to be stationary. Otherwise, a non-rigid ICP algorithm, which registers a pair of point clouds, can be used for reconstructing 3D geometry of a non-rigid object directly, but it often fails due to the ambiguity in point correspondences. This paper presents a method for generating a 3D shape template from a single RGB-D sequence. In order to reduce the ambiguity in point correspondences, our method leverages point trajectories obtained in the RGB images, which can be used for associating points in different point clouds. We demonstrate the capability of our method using deforming human bodies. Introduction Recently, various applications that present a moving and deforming object to users, such as virtual fitting room [1] and virtual pets [2], have become available to ordinary users. These applications render objects based on 3D geometry of their entire shapes (which we refer to as full-body shape models) and their non-rigid motion, both of which are usually handcrafted. Automatic techniques for reconstructing full-body shape models at each frame can drastically reduce the cost for creating a 3D shape model and motion (e.g., [3, 4]). They use multiple sensors (e.g., RGB or RGB-D sensors), whose relative poses are known, to capture the object from different viewpoints simultaneously. It then applies an existing 3D reconstruction technique for rigid objects, such as [5, 6, 7]. However, the use of multiple sensors may be still cumbersome for some applications in which ordinary users need their own shape models and motions. Reconstructing 3D shape and motion from a single sensor is a challenging problem. Two approaches have been proposed: one uses 3D shape templates of the target object and the other does not. Former approach [8, 9] generates a 3D shape template using a 3D shape reconstruction technique for rigid objects [5, 6, 7], assuming the object is almost stationary. They then fit the 3D shape template to a 3D point cloud at each frame of a single depth map sequence. One major limitation of this approach is that it requires an extra burden to capture the target object while it is stationary, which is practically infeasible, especially for objects like animals. Latter approach [10, 11] registers 3D point clouds in all frame of a single depth map sequence to any other frames using non-rigid iterative closest point (ICP) [12, 13]; however, it often 3D shape template